Skip to content

docs: cover example solutions and check design for scorers - #205

Open
czenko wants to merge 1 commit into
mainfrom
mirandalimonczenko/docs-1324-document-example-solutions-and-check-design-in-the-evals
Open

docs: cover example solutions and check design for scorers#205
czenko wants to merge 1 commit into
mainfrom
mirandalimonczenko/docs-1324-document-example-solutions-and-check-design-in-the-evals

Conversation

@czenko

@czenko czenko commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Closes DOCS-1324

Problem

Ten more docs evals are scoped. The RLS eval took several review rounds, and most of the findings were scorer defects a contributor can catch before asking for review. CONTRIBUTING.md doesn't say how. Guidance comes from the review of #168.

Solution

  • Adds Checking your scorer: one green solution that passes every check, a few bad ones carrying one deliberate flaw each, and the expected failures written down before scoring. Split into solution shapes, what to write, how to score, and reading the results.
  • Names three solution shapes, since the 44 evals don't share one. A workspace overlay for local-stack evals, a remote/-shaped seed for tools evals, and a report for the 8 evals scored on what the agent said rather than what it built.
  • Adds Designing checks: decide what a check does when the object is absent, and find a class of object by asking the system that owns it. Checks about an object the prompt names can name it.
  • Notes that a flaw usually trips several checks, and that transcript checks can't be exercised by a solution you wrote yourself.
  • Lists solutions/ in the eval folder contents, above the optional entries, and links the new section from Submitting evals for review.

Manual testing

  1. Open CONTRIBUTING.md on this branch. Designing checks and Checking your scorer sit between Writing scorers and Adding an experiment.
  2. Click either Checking your scorer link in that page. Both jump to the new section.
  3. Read the diff of the new content. See that it is technically accurate and helpful for all Eval contexts.

@vercel

vercel Bot commented Aug 14, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
evals Ignored Ignored Preview Aug 14, 2026 7:31pm

Request Review

@czenko
czenko marked this pull request as ready for review August 14, 2026 19:28
@czenko
czenko requested a review from a team August 14, 2026 19:28
Guidance from the review of #168, generalized past the RLS eval so it
holds for all 44 scenarios.

- Checking your scorer: one green solution, a few bad ones, and the
  expected failures written down before scoring.
- Solution shapes: a workspace overlay, a remote-shaped seed, or a
  report, depending on what the eval measures. Eight evals score
  nothing but the report.
- Designing checks: what a check does when the object it inspects is
  absent, and finding a class of object from the system that owns it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@czenko
czenko force-pushed the mirandalimonczenko/docs-1324-document-example-solutions-and-check-design-in-the-evals branch from 3a9e5d1 to 66ed0da Compare August 14, 2026 19:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant